Skip to content

docs: add an automated access guide for service accounts - #37

Draft
alukach wants to merge 2 commits into
mainfrom
docs/unattended-workflows
Draft

docs: add an automated access guide for service accounts#37
alukach wants to merge 2 commits into
mainfrom
docs/unattended-workflows

Conversation

@alukach

@alukach alukach commented Sep 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

A new page, Automated Access (/automated-access, last under Using Source), for software that reaches Source Cooperative with nobody at the keyboard. It covers what a service account is; creating one and granting it products, as the UI flow; issuing an API key (shown once) and the five variables the AWS CLI or an SDK needs to exchange it at the data proxy and refresh on its own, with AWS CLI and boto3 examples and their minimum versions; the refusal a revoked, expired or unknown key gets and the request id to quote to support, and the separate error for a key cut short or mistyped, which its checksum catches; keeping the key out of URLs and aws --debug output; several keys per service account for rotation, and Change expiry; tools that keep their first credentials, so long transfers fail mid-flight; GDAL; revocation and disabling as the emergency stop, with their timings; revoking a key you found; and GitHub Actions as a clearly marked "coming soon".

Upload Your Data's Option 3 walked through trusting an IAM role in the uploader's own AWS account to write straight to Source's bucket. Service accounts supersede that flow, so the section is now a short pointer to the new page, as are the two earlier references to it on that page and the "contact us" line for automated access. The heading is kept as it was so #option-3-longstanding-or-automated-access-advanced still lands. Access Data gains one sentence pointing unattended software at service accounts.

This documents features whose code is still in open PRs, so it merges only after they deploy: source-cooperative/source.coop#570 (key lifecycle), source-cooperative/source.coop#580 (opaque keys, the variables printed at issue, Change expiry, the Disable wording), source-cooperative/source.coop#581 (the public self-revoke route), source-cooperative/source.coop#596 (keys that end in a checksum), source-cooperative/data.source.coop#235 (the proxy's key exchange), and source-cooperative/data.source.coop#221 (the FullAccess and ReadOnly role names).

Decisions to flag

  • Role ARN and region. The page uses AWS_ROLE_ARN=arn:aws:iam::<service-account-id>:role/FullAccess and AWS_REGION=us-west-2, the form the issue dialog in feat(accounts): opaque API keys resolved by hash, and a working key UI source.coop#580 prints and the GitHub snippet on source.coop main already uses. It documents FullAccess as everything the service account may do and ReadOnly as reads only. Both names need Add the ReadOnly Role alongside FullAccess data.source.coop#221, which is being built; until then the proxy serves only _default, which stays as an alias of FullAccess and which the page doesn't mention.
  • GDAL sends the key to AWS, not to the proxy. The brief said GDAL's own STS call puts the token in the URL, which the proxy refuses. GDAL's source (port/cpl_aws.cpp, checked at 3.6.0, 3.12.0 and master) shows it is worse than that. GDAL never reads AWS_ENDPOINT_URL_STS; its STS root is CPL_AWS_STS_ROOT_URL, defaulting to https://sts.<AWS_REGION>.amazonaws.com. So on a machine with the five variables and no other credentials, GDAL 3.6 or later sends the key to AWS in a GET query string. The page says to set CPL_AWS_WEB_IDENTITY_ENABLE=NO (present since 3.6.0) wherever GDAL runs beside the variables, and to replace a key GDAL has already seen. The coming-soon note for the Source CLI (Unattended refresh: exchange an API key without a browser and keep the token file fresh source-coop-cli#17) names GDAL 3.12, the first release that reads credential_process and refreshes it on expiry.
  • Tools that keep their first credentials. The issue says GDAL, DuckDB and rclone "capture environment variables at process start". The page puts it as fixed credentials (the three AWS_* values, or the same values in a tool's own settings), because that is where the mid-transfer failure is certain. GDAL 3.12+ with credential_process refreshes, and so can recent DuckDB. The failure is named as the proxy returns it: ExpiredToken, HTTP 403 (multistore 0.7.2).
  • DuckDB's credential_chain isn't recommended with a key. The page says DuckDB reads a secret's credentials at CREATE SECRET and suggests CREATE OR REPLACE SECRET between batches. It doesn't suggest PROVIDER credential_chain with the five variables. Whether DuckDB's bundled AWS C++ SDK sends that exchange to AWS_ENDPOINT_URL_STS depends on its version: the older internal STS client hard-codes sts.<region>.amazonaws.com, while the CRT-based provider reads an endpoint override. If it doesn't, the key goes to AWS. Verifying that is Verify an unmodified AWS SDK acquires and refreshes credentials from the environment alone data.source.coop#229's work. rclone likewise appears only in the fixed-credentials framing.
  • Old SDKs. Releases before AWS CLI 2.13.0 and botocore 1.31.0 (boto3 1.28.0) ignore the endpoint variables and send the key to AWS. The page says so; the versions come from the three projects' changelogs.
  • Lifetimes. "An hour by default, up to 12 hours if the client asked" comes from the proxy's 3600-second default and STS_MAX_SESSION_DURATION_SECS = "43200" in production on feat(sts): exchange opaque API keys at /.sts by hash lookup data.source.coop#235's branch.
  • Self-revoke and secret scanning. Revoking a found key with POST https://source.coop/api/v1/service-account-keys/revocations lands with feat(accounts): revoke leaked API keys via self-revoke and GitHub secret scanning source.coop#581. It takes the key in a JSON body and answers 204 for any well-formed key. Automatic revocation of keys pushed to public GitHub also lands with feat(accounts): revoke leaked API keys via self-revoke and GitHub secret scanning source.coop#581, but it needs GitHub partner registration, so the page says only that it is planned.
  • GitHub Actions is a short "coming soon" note, not instructions, because trusted workflows need Namespace the credential subject by verified issuer data.source.coop#222 and Trust GitHub Actions alongside the Source issuer data.source.coop#223. It warns that the service account page already offers adding a workflow, and says a workflow can use an API key meanwhile.
  • Left in place: the IAM and bucket policy wizards. /tools/iam-policy-wizard and the internal /tools/bucket-policy-wizard served the superseded Option 3 flow. Nothing in the docs links to them now, and the IAM wizard's intro still sends readers to Upload Your Data "for the full walkthrough". Whether to retire them depends on whether anyone is still onboarded through IAM roles, so that is a separate change.
  • Scope against the issue text. This follows the revised scope for Unattended workflow guide #34 (not yet applied on GitHub) where the issue text differs. It covers the server, VM and HPC path, with one line for cron and systemd. It has no Kubernetes snippet, since the five variables don't change by environment. GitHub Actions is "coming soon" rather than documented, and the SDK path doesn't wait on Unattended refresh: exchange an API key without a browser and keep the token file fresh source-coop-cli#17; only GDAL does. The Access Data sentence goes beyond the brief; it is there because that page is where readers wanting programmatic access land.

How I tested it

  • docusaurus build (Docusaurus 3.10.2, Node 22.9.0) succeeds with the site's onBrokenLinks: 'throw', and prints no broken-link or broken-anchor warnings. I built after the final edits and checked the output: the page renders at /automated-access, the sidebar lists it after Access Data, Access Data's "Next" goes to it, #github-actions resolves, both admonitions render, and the Option 3 anchor still exists. npm ci in the worktree failed with ENOSPC (the disk is nearly full), so the build ran against the main checkout's existing pnpm install through a symlink, with DOCUSAURUS_NO_PERSISTENT_CACHE=1 so nothing was written into it. I removed the symlink and the build output afterwards.
  • Not run: the page's commands against a live proxy. That needs the PRs above deployed.
  • I checked the facts against their sources: GDAL's source at 3.6.0 through 3.12.0 and master; the AWS CLI v2, botocore and boto3 changelogs; multistore 0.7.2's error mapping; feat(sts): exchange opaque API keys at /.sts by hash lookup data.source.coop#235 for the refusal messages and the request id (the Cloudflare ray id); and the UI labels on source.coop main and on feat/opaque-api-keys at c85aff23.

Docs and ADRs

This is the docs half of source-cooperative/source.coop#570, source-cooperative/source.coop#580, source-cooperative/source.coop#581 and source-cooperative/data.source.coop#235. I checked ADR-013 as revised (source-cooperative/data.source.coop#234) and ADR-014 (source-cooperative/data.source.coop#232). The page describes what they decide: the five variables, a key only in a request body, one refusal carrying a request id, revocation within the 60-second cache, disabling as the emergency stop (writes within a minute, restricted reads within five), and FullAccess/ReadOnly. Neither needs a change. The other Using Source pages (Create an Account, Create a Data Product, Bring Your Own Bucket) still hold.

Related

Part of #34. It isn't Closes: the GitHub Actions section and the GDAL setup wait on source-cooperative/data.source.coop#222, source-cooperative/data.source.coop#223 and source-cooperative/source-coop-cli#17. #32 ("Document unattended credential acquisition") covers the same ground and looks like a duplicate of #34; I left it untouched. Part of source-cooperative/source.coop#491.

🤖 Generated with Claude Code

https://claude.ai/code/session_01R1eiTse4416N6uTgAy4Ddd

@alukach @claude
docs: add an automated access guide for service accounts
e6d9afc 
Adds Automated Access (docs/using-source/automated-access.md), the guide for software that reaches Source Cooperative with nobody at the keyboard: what a service account is; creating one and granting it products; issuing an API key and the five variables the AWS CLI or an SDK needs to exchange it at the data proxy; the refusal users see and the request id to quote; keeping the key out of URLs and debug output; rotation and editable expiry; tools that keep their first credentials; GDAL; revocation, the emergency stop and revoking a found key; and GitHub Actions as coming soon. The page is added to sidebars.ts.

Upload Your Data's Option 3 walked through trusting an IAM role in the uploader's own AWS account to write straight to Source's bucket. Service accounts supersede it, so the section is now a pointer to the new page, as are the two references to it earlier on that page and the "contact us" line for automated access. Access Data gains one sentence pointing unattended software at service accounts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R1eiTse4416N6uTgAy4Ddd
@vercel

vercel Bot commented Sep 25, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs-source-coop Ready Ready Preview Sep 29, 2026 11:31pm UTC

Request Review

@alukach alukach mentioned this pull request Sep 29, 2026
Merged
alukach added a commit to source-cooperative/source.coop that referenced this pull request Sep 29, 2026
@alukach @claude
feat(accounts): issue, revoke and expire opaque API keys for service …
2bc0480 
…accounts (#570)

Closes #548. Closes #546. API keys are the second Integration type
beside the GitHub trusts #567 shipped, and #566 replaced proof of
control with per-account trust. The `source.coop` half of #548. Part of
#491.

**Merge order:** this can merge and deploy before the proxy. It no
longer calls the proxy; the proxy calls it.
source-cooperative/data.source.coop#235 is the proxy half and needs the
route here to be live, or every key exchange fails closed.

## What

API keys for environments without OIDC — a server, a scheduler, an
instrument — per ADR-013 as revised in
source-cooperative/data.source.coop#234: **a key is an opaque secret
that source.coop resolves by hash; nothing signs it.** Six commits on
`main`: the original feature, dropping the Ory-id guard once #567
namespaced service-account ids, #580's rework to opaque keys, the key
hint, a calmer key list, and that list as its own component with
stories.

**The key** is `sck_` + 32 random bytes in base64url: a fixed 47
characters, all entropy after the prefix, matching
`sck_[A-Za-z0-9_-]{43}`, which is what gets registered with secret
scanners (#561). It is shown once and never stored.

**The record** (`service-account-keys` table) is keyed by the key's hex
SHA-256, with a public `key_id` (a UUID) for listing, revoking and
changing expiry, a label, who issued it and when, an optional expiry,
`revoked_at` and `last_used_at`. `publicKey()` strips the hash before
any record reaches a client component.

**The hint**: the record also keeps the key's last four characters, and
the key list shows each key as `sck_…Xy9Q`, so someone holding a key can
tell which record, and so which service account, it is. Four characters
are 24 of the key's 256 random bits, leaving far too many to guess. The
hint only confirms a key in hand against its record; it is never used to
look a key up, since keys across the platform will share it. The issue
dialog says how the new key will be listed. Keys issued before the hint
existed are listed without one.

**Issuing** (`issueApiKey`): whoever manages the service account gives a
label and an expiry (30/90/365 days, or never). The action generates the
key, writes the record and returns the key once. A disabled service
account is refused. **Revoking** sets `revoked_at`; **expiry** can be
changed after issuance, including to never.

**Exchanging**: `POST /api/v1/service-account-keys/exchanges` with
`{key_hash}` is what the proxy calls at `/.sts` when a key is presented,
authenticated as the proxy itself (`verifyProxyAssertion`, sentinel
subject `urn:source:data-proxy`, recorded in the ADR-005 amendment in
source-cooperative/data.source.coop#234). It always answers 200 with
`{account_id, key_id, active}`. `active` means known, not revoked, not
expired, and its service account not disabled; an unknown hash is
answered as inactive, indistinguishable from a revoked one. It records
last use. The proxy caches the answer for 60s, so revocation takes
effect for new exchanges within that time; credentials already issued
live to their session cap.

**Resolving the subject**: after an exchange, the proxy's credentials
name the service account itself, and `authenticateWithOidcToken`
resolves it with `fetchByOryId`, then a service account by id. No
service account's id can be someone's Ory identity id: #567 namespaces
it as `{owner}--{id}`, and a UUID never contains `--`.

**UI**: the service account's page (#567's `ServiceAccountDetail`) has
an API keys section, rendered by `ApiKeyList`, a row per key. On the
left, the label with a Revoked or Expired marker, and the key's hint
beneath. On the right, two short lines: how it has been used ("Used 3
days ago", "Never used") and when it ends ("Expires in 5 months", "Never
expires", "Revoked 9 months ago"). The exact dates, and who issued the
key and when, are in their tooltip. Change expiry and Revoke are in a
"⋯" menu; a revoked key has none, and keeps an invisible copy of the
button so its lines align. The section header carries
`IssueApiKeyDialog`, which shows the key once with a copy button, and
the environment variables that point any AWS SDK or the AWS CLI at the
proxy. The list row counts live keys as a way to sign in.

![The API keys section: HPC cron job over sck_…Xy9Q, with Used 6 months
ago and Expires in 5 months on the right and a ⋯ menu; Old laptop,
marked REVOKED, over sck_…a_7k, with Never used and Revoked 9 months
ago](https://raw.githubusercontent.com/source-cooperative/source.coop/assets/service-account-management/service-account-key-rows.png)

Every state a key can be in, from `ApiKeyList`'s Default story:

![Five keys: HPC cron job, used 3 days ago, expires in 5 months;
Instrument uploader, used today, never expires; Laptop, for testing,
never used, expires next month; Last year's sync marked EXPIRED, used
last month, expired last month; Old laptop marked REVOKED, never used,
revoked 9 months ago. Each shows its sck_… hint; live keys have a ⋯
menu](https://raw.githubusercontent.com/source-cooperative/source.coop/assets/service-account-management/api-key-list-states.png)

## Stories

- `ApiKeyList` › **Default** (every key state), **Single**,
**WithoutHint**, **Empty** —
https://source-coop-ui-git-feat-service-account-keys-radiantearth.vercel.app/?path=/story/features-service-accounts-apikeylist--default
(dates are set relative to today; hover a row's dates for the exact
ones)
- `ServiceAccountDetail` › **Default** —
https://source-coop-ui-git-feat-service-account-keys-radiantearth.vercel.app/?path=/story/features-service-accounts-serviceaccountdetail--default
- `ServiceAccountList` › **Default** —
https://source-coop-ui-git-feat-service-account-keys-radiantearth.vercel.app/?path=/story/features-service-accounts-serviceaccountlist--default
- `IssueApiKeyDialog` › **Default** —
https://source-coop-ui-git-feat-service-account-keys-radiantearth.vercel.app/?path=/story/features-service-accounts-issueapikeydialog--default
(submitting reaches the show-once view with the hint line)
- `ApiKeyExpiryField` —
https://source-coop-ui-git-feat-service-account-keys-radiantearth.vercel.app/?path=/story/features-service-accounts-apikeyexpiryfield--default

## Testing

- `npx jest` — 79 suites, 835 tests, all pass on the branch as rebased
onto `main`. `service-account-keys.test.ts` covers issuing (the record
holds the hash, never the key; the hint is the key's last four
characters; the key is returned once), no-expiry keys, refusals before
any write, revoking only own keys, and expiry changes including to
never. The exchanges route test covers the proxy-only auth, active and
inactive answers, and last-use recording.
- `npm run type-check`, `next lint`, `npm run build-storybook` — clean.

## Docs and ADRs

data.source.coop: this implements ADR-013 as revised in
source-cooperative/data.source.coop#234, with ADR-014 from
source-cooperative/data.source.coop#232; the proxy side is
source-cooperative/data.source.coop#235, which supersedes
source-cooperative/data.source.coop#233. The hint is a display detail
the ADR doesn't need. docs.source.coop: the unattended-workflow guide,
source-cooperative/docs.source.coop#37 for
source-cooperative/docs.source.coop#34, should mention matching a key to
its account by its last four characters.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
@alukach @claude
docs: say what the malformed-key error means
459c6aa 
An API key now ends in a six-character checksum of the rest (ADR-013, source-cooperative/data.source.coop#242), so the data proxy and the Source CLI refuse a key that was cut short or mistyped before anything is looked up, with "API key is malformed; check that it was copied whole". The guide's "If the key is refused" section shows that error and what to do about it, and no longer says a value that isn't a key gets "API key was not accepted".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R1eiTse4416N6uTgAy4Ddd
@alukach alukach mentioned this pull request Sep 29, 2026
Open

This branch was successfully deployed

1 active deployment
Preview — 459c6aae Deployed Sep 29, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@alukach